decode an byte buffer as a variable sized integer of the given type (signed or unsigned).