← All Modules / Module 2 / Session 2.1 / KP3

The Selection Problem

Discover why Bengio's fixed concatenation approach fails for dynamic language understanding

🎯 Interactive Demonstration

Select an example below to see how Bengio's fixed window approach misses important context.

Bengio's Fixed Window Size

2 6 4

Adjust the window size to see how it affects what information is available for prediction.

In Window: Used for prediction
Outside Window: Ignored
Target: Word being predicted
Important but Missed: Should matter but outside window

Select an example above to see the selection problem in action.

❌ Bengio's Fixed Concatenation
  • Fixed Window: Always uses last N words
  • Equal Weights: Every word gets same importance
  • Position-Based: Location determines relevance
  • Context Blindness: Can't see beyond window
Core Problem: Important words outside the fixed window are completely ignored, leading to poor predictions when crucial context is distant.
✅ What We Need Instead
  • Dynamic Selection: Focus on relevant words anywhere
  • Content-Based: Importance based on meaning
  • Flexible Context: No fixed window limitations
  • Smart Weighting: Important words get more attention
The Solution: Attention mechanism (Session 2.2) provides dynamic, content-based selection that can focus on any relevant word in the sequence.