Cardiovascular disease (CVD) remains a global health priority. While administrative claims data offer a longitudinal view of patient history, their high dimensionality and extreme class imbalance make traditional risk prediction difficult. This study evaluates transformer-based architectures (BERT, BioBERT, and ClinicalBERT) for one-year CVD event prediction using a nationwide health registry, the French National Health Data System (SNDS). Using a cohort of 10.7M individuals, we compared the performance of these models against a Random Forest (RF) baseline. While all models showed high accuracy (> 98%) due to low event prevalence (1.2%), domain-adapted transformers (BioBERT and ClinicalBERT) significantly outperformed RF in clinical utility, achieving an F1-score of 16.8% and a 15-fold increase in recall. These results show that although transformer models capture some longitudinal information, overall predictive performance is modest, likely due to the loss of information when structured claims data are converted into text format for transformer models.