During the collision between the coming down floors and the next standing one, the weights of the objects involved are almost irrelevant for all practical reasons (in the first few floors crashing, when speed is still not very big, its not trivially unimportant, but the next ones make it really tiny to irrelevant). Momentum is the important thing.
The collision takes a fraction of the second (to break the floor) and the forces generated are many times that of the weight that floor used to support all these years. This is why you can jump on a board and break it on the way down but if you just carefully stand on it, it will hold you just fine. You can stand on top of a car and it wont bend but if you jump up and down you can create a bad dent. So its not your weight that does it, its the momentum you have on impact, the contact interactions create that force, during deceleration, that crashes the surface of contact. Read about eg impulse here;
http://en.wikipedia.org/wiki/Impulse_%28physics%29. Then recall what it feels to get hit by a baseball ball from a pitcher etc (the weight of the ball is almost irrelevant).
The floor below upon contact with the upper floors ,that were dropping, starts reacting and decelerating the upper floors by exerting forces on them and at the same time it feels the same forces opposite stressing it until it crashes when the maximum level it can withstand is exceeded. These forces in fact exceed the strength of the material by many times and they crash it easily, without even more than a tiny fraction of the second managing to produce a deceleration (resistance to crashing). Have no doubt that during that fraction of the second the floors falling do indeed also feel a force many times their weight but it is very briefly so their overall drop is not significantly inhibited. Obviously all these collision surfaces are locally destroyed/broken in pieces as it all falls but the majority of the building above that area is still intact, not broken as it falls.
Try it. Is the collision less than 0.01sec? Then for momentum m*v of falling floors to even change by 5% you would need impulse (the integral of F over time) to be 0.05*m*v so the avg F (Impulse~F*Dt) is like order 0.05*m*v/0.01= 5*34*10^6*(2*9.81*3.8)^(1/2)~1.5 *10^9 Nt which is like;
0.05/0.01(2*9.81*3.8)^(1/2)/9.81~4.4 times the weight of the building at the first crashing floor level!
See what i did here? I basically told you that if the force acts 0.01s and takes off only 5% of the momentum the system has, it will feel as if 4.4x the weight. Massive. Reduce the momentum by 10-20% and you get 10-20 times the weight etc. And you still havent done a lot to stop it from falling (20% is still little). This is why i said the weights are irrelevant during the collision. The forces created on impact are many times larger. This is why it crashes so easily. Its like a car collision accident.
Can it be more than 0.01-0.02sec during collision per floor? Not really. Because 100*0.02=2sec and the free fall from say 325-375m avg height of the destroyed floors is like 9 seconds and the entire fall is like 10-11 sec or so, so its cant be more than 0.02sec as you have to do it for nearly 80-90 floors one by one and it has to remain below 10sec or so total.
When each building floor is designed to hold say a bit over the weight of the floors above, say eg 1.5x that weight (to leave some room for safety), the collision forces it to experience many times that. It is impossible for it not to break and join the collapse. I am not sure what tolerance they have given it but it cant be much more than 1+ something little, because its not needed for anything big ever ( ie more than earthquakes and other fluctuations or an occasional building accident/local failure etc) other than now and even now it wont matter how big the tolerance is, if the collapse happens at lower than 100-105th floor or something like that, its game over anyway, it cant stop it.
Now obviously i am simplifying things to make them easier to understand at a practical working level. The real effect is much more complicated and its chaotic in certain sense as breaks are taking place all over the structure and a floor is never uniformly falling to another floor, its always a bit asymmetric, some part feels the forces first etc. But the main idea is not going to change. Once the drop has started and the mass of the floors falling is significant, the forces generated upon contact of each new floor with the falling floors are easily exceeding the strength of the materials of that floor and the break happens one floor after another with even easier manner and this is why it looks like free fall. It delays only slightly with each collision and the speed continues to increase as the forces experienced last only a tiny fraction of the second each floor, never really reducing the speed gained all that much, until another 3.8m of floor height gives in to lead to more acceleration again and a new collision with even greater momentum to follow, repeating the cycle with advanced ferocity.