It’s Oliver’s look at the end that gives him away. At the end he doesn’t even scramble to explain. He knows he’s disappointed Felicity hugely and even though her hurt is muted, he seems to really take it and then makes a face of frustration at himself. He didn’t want to hurt or disappoint her, and Oliver Queen the playboy wouldn’t have cared. But Oliver Queen the man who values Felicity’s opinion cares a great deal and her subtle disappointment is more effective at gutting him than if she’d gotten angry.