An extensive and diverse dataset is a key requirement for successful training of a deep model. Compared to on-site data collection, 3D modelling allows to generate large datasets faster and cheaper. Still, the diversity and the perceptual realism of synthetic images remain in the realm of the experience of a 3D modeller. Moreover, hard sample mining with 3D modelling poses an open question: which synthetic images are challenging for an object detection model? We present an Adversarial 3D Modelling framework for training an object detection model against a reinforcement learning-based adversarial controller. The controller alters the 3D simulator parameters to generate complex synthetic images. The aim of the controller is to minimize the score of the object detection model during the training time. We hypothesize, that such objective of the controller allows to maximize the score of the detection model during inference on real-world data. We evaluate our approach by training a YOLOv3 object detection model using our adversarial framework. A comparison with a similar model trained on random synthetic and real images proves that our framework allows to achieve better performance than using random real of synthetic data.