In all honesty, I don't think that there is a downside to informal assessment. All teachers work out of some type of informal assessment framework. Even school districts that are heavily driven by standardized assessment still have to rely on teachers' informal assessment for guidance. Informal assessment provides a real time read on where students are on a daily basis. They can be in so many different forms that so much falls under the informal assessment umbrella. Homework can be informal assessment. Exit slips or even a "thumbs up" or "thumbs down" approach to instructional aims are both examples of informal assessment. Even discourse is a form of informal assessment. If I stop a student in the hallway and ask them about connecting themes of industrialization to the brick pattern in a wall, the dialogue that ensues can be seen as a form of informal assessment. Informal assessment has the added benefit of being relatively less stressful than standardized testing because the assessor, the teacher, is more known to the student. Sometimes, informal assessment can have the added benefit of students not even knowing they are being "assessed," such as with project work and or anecdotal forms of informal assessment.
If there is a downside to informal testing, it is in the fact that it does not appear to be as validated as high stakes standardized testing. Formal assessment such as standardized testing and computer based external assessment are credited as being "Data driven" means in many districts. This helps to discredit informal assessment. Formal modes of assessment are in the process of being more widely accepted by school districts and administrators. This does not illuminate a downside to informal assessment or testing. Yet, it does show how it is being set aside in favor of more external and formal assessments in which teachers are more proctors than actual participants in the assessment process.