You're admitting you have a passphrase, and therefore that you have access to the information it protects. This piece of information is non-trivial, as it eliminates one avenue of plausible deniability, and actions that implicitly provide this information are recognized as testimonial in nature.