ido-work-directory-match-only

ido-work-directory-match-only is a variable defined in `ido.el'.
Its value is

t

Documentation:
Non-nil means to skip non-matching directories in the directory history.
When some text is already entered at the `ido-find-file' prompt, using
M-x ido-prev-work-directory or M-x ido-next-work-directory will skip directories
without any matching entries.

You can customize this variable.