High Scalability Building bigger, faster, more reliable websites.

Thanks for your suggest !
Yes, each document is a brief description and 1 ID, so a single document is very light.

With my initial solution : 1 search job == all results
The search job is executed 1 time, and the paging system show only parts of the results.
The problem is : 25 document * X pages * Y users may be huge.

With your solution : 1 search job == results for 1 page.
One search job is executed for each page view.
In term of storage, your solution is good and lightweight.
But the risk is : navigation between page may be slow because it generate more jobs.

In term of CPU, according to the lucene FAQ :
* How do I implement paging, i.e. showing result from 1-10, 11-20 etc?
-> Just re-execute the search and ignore the hits you don't want to show. As people usually look only at the first results this approach is usually fast enough.

Each page view will generate the same job to the lucene engine.
It may be slower for the user experience and more "cpu intensive" for the job search workers.

I will try an intermediate approach :

1 job search will generate results for 5 pages.
Each search result will be cached (memcache).
When user ask for page 4 it wil generate a new search job in background for page 5-10 (if not already in cache of course).

In term of cpu and memory usage, it will be lighter than the 2 previous solutions.
What do you think about it ? Do you see any other possible improvements ?
Do you think i'm paranoid in term of storage ? ;)