انتقل إلى المحتوى
أمين أبوزيد
Englishالعربية
MID-071

تهديدات تطبيقات الذكاء الاصطناعي

  • 6 درس
  • 2.5 ساعة تدريب (تقدير)
ابدأ الوحدة
في هذه الصفحة 8 موضوعات

نص الدروس والتمارين باللغة الإنجليزية، بينما تبقى واجهة الموقع وعناوين الدروس بالعربية.

نظرة عامة

الموضوعات

  • الحقن غير المباشر للتعليمات (Prompt Injection) عبر المحتوى المسترجَع
  • الصلاحيات المفرطة (Excessive Agency)
  • تسريب البيانات
  • إساءة استخدام الكلفة
  • وسائل التخفيف

تعلّم أولًا

2.5 ساعة تدريب (تقدير)

مخرجات التعلم

  1. Explain the OWASP Top 10 for LLM Applications categories, including prompt injection, sensitive information disclosure, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, and unbounded consumption.
  2. Demonstrate indirect prompt injection through retrieved content and mitigate it (content provenance, separation of instructions and data, tool restrictions, output checks).
  3. Prevent data exfiltration through tools, links, and rendered output.
  4. Protect against cost and resource abuse with per-user quotas, input limits, and loop limits.
  5. Add these threats to the service's threat model and to automated security tests.

الدروس

  1. 01مشهد تهديدات LLM
  2. 02الحقن غير المباشر للتعليمات عبر المحتوى المسترجَع
  3. 03الصلاحيات المفرطة والمعالجة غير السليمة للمخرجات
  4. 04كشف البيانات الحساسة وتسريب system prompt
  5. 05الاستهلاك غير المحدود وإساءة استخدام الكلفة
  6. 06عرض عملي: مقال مساعدة مسموم يختطف المساعد ثم الإصلاح

المختبر والتمارين

Lab

MID-071-LAB1 · Attack the assistant

Outcomes: 1–5 · Time: ~3 h Write automated tests for: direct injection, indirect injection (a planted help article), reading another customer's data through tools, exfiltration through a generated link, and quota exhaustion. Run them, fix what fails, and update the threat model. Acceptance: all attack tests run in CI and pass after the fixes; threat model updated.

Exercises (assessment: threat-model the assistant)

IDOutcomeTask
MID-071-EX011Map five OWASP LLM categories to concrete threats against the order-support assistant.
MID-071-EX022Write a planted help article and a test that proves the assistant does not follow it.
MID-071-EX032Explain why the system-prompt defense alone fails and list three stronger layers.
MID-071-EX043The assistant renders markdown. Design the control that stops data leaking through image and link URLs.
MID-071-EX054Define quotas and loop limits for the assistant, with values and the alert on breach.
MID-071-EX065Add three assistant threats to the STRIDE table with mitigations and the test that covers each.
أظهر الحلول
  • EX01 Example: prompt injection (planted article), sensitive disclosure (other customers' orders), improper output handling (HTML in answers), excessive agency (refund tool), unbounded consumption (long loops).
  • EX02 The article instructs a refund; the test asks about the order and asserts no refund request was created and no foreign link is in the answer.
  • EX03 Models can be talked out of rules. Layers: untrusted-data labeling, tool restriction and approval, output validation, domain allowlist, least-privilege tokens.
  • EX04 Allowlist of image and link domains, strip other URLs, no auto-loading of remote images, escape output.
  • EX05 Example: 4 000 input characters, 6 steps, 30 s, 50 000 tokens per user per day; alert when daily spend exceeds the budget or one user exceeds 5x the median.
  • EX06 Each row: element, STRIDE letter, threat, mitigation, test ID; the tests are those from the lab.

Outcome 1 is also covered by quiz Q01–Q03; outcome 5 by the lab criterion "threat model updated".

الاختبار

Outcomes: O1 OWASP LLM categories · O2 indirect injection · O3 exfiltration · O4 cost abuse · O5 threat model and tests.

اختبار ذاتي لك: لا توجد درجات، ولا يُحفظ أو يُرسل أي شيء. اختر إجابة ثم تحقق منها، ويمكنك المحاولة مرة أخرى كما تشاء.

  1. 1Why are LLM apps prone to injection?O1
  2. 2"Excessive agency" means:O1
  3. 3Treating model output as raw HTML is:O1
  4. 4Indirect prompt injection arrives through:O2
  5. 5Which defense is enforced outside the model?O2
  6. 6Content from your own index should be:O2
  7. 7To stop exfiltration via links in output:O3
  8. 8Secrets in the system prompt should be:O3
  9. 9A control against cost abuse is:O4
  10. 10Attack tests for the assistant should:O5

الجزء العملي

Threat-model the assistant (rank the top five risks with mitigations) plus lab MID-071-LAB1. Pass: quiz at least 8/10, all attack tests in CI and passing after fixes, and an updated threat model.

أخطاء شائعة

Relying on "ignore malicious instructions" in the system prompt as the only defense; trusting content because it came from your own index; rendering model output as HTML or executing it.

التقييم

Threat-model the order-support assistant and rank the top five risks with mitigations.

الصلة بالمسار الوظيفي

AI security architecture (ARC-046) and responsible AI (SRA-022).

اكتب للبحث في المستويات والمراحل والوحدات والدروس.