Are you sure you have the pins connected right? If you have the tx pin from device connected ok you could see the output, but if the rx pin is wrong then no giving it commands.
From the bottom of the header its:
GND...... NOPIN...... TX...... RX....... 3.3V
I have a usb keyboard as well so doubt its that.
Edit: should specify, you should have no need to connect to the 3.3V if your converter is already powered over USB.
Edit2: HA! I got the TX RX pins swapped, my bad its:
GND...... NOPIN...... RX...... TX....... 3.3V
From the modem, the connections to your serial adapter are obviously swapped.