| | --- |
| | license: llama2 |
| | datasets: |
| | - AlfredPros/smart-contracts-instructions |
| | language: |
| | - en |
| | tags: |
| | - code |
| | - blockchain |
| | - solidity |
| | - smart contract |
| | --- |
| | # Code LLaMA 7B Instruct Solidity |
| |
|
| | A finetuned 7 billion parameters Code LLaMA - Instruct model to generate Solidity smart contract using 4-bit QLoRA finetuning provided by PEFT library. |
| |
|
| | # Training Dataset |
| |
|
| | Dataset used to finetune the model is AlfredPros' Smart Contracts Instructions (https://huggingface.co/datasets/AlfredPros/smart-contracts-instructions). |
| | A dataset containing 6,003 GPT-generated human instruction and Solidity source code data pairs. This dataset has been processed for training LLMs. |
| |
|
| | # Training Parameters |
| |
|
| | ## Bitsandbytes quantization configurations |
| | - Load in 4-bit: true |
| | - 4-bit quantization type: NF4 |
| | - 4-bit compute dtype: float16 |
| | - 4-bit use double quantization: true |
| |
|
| | ## Supervised finetuning trainer parameters |
| | - Number of train epochs: 1 |
| | - FP16: true |
| | - FP16 option level: O1 |
| | - BF16: false |
| | - Per device train batch size: 1 |
| | - Gradient accumulation steps: 1 |
| | - Gradient checkpointing: true |
| | - Max gradient normal: 0.3 |
| | - Learning rate: 2e-4 |
| | - Weight decay: 0.001 |
| | - Optimizer: paged AdamW 32-bit |
| | - Learning rate scheduler type: cosine |
| | - Warmup ratio: 0.03 |
| |
|
| | # Training Details |
| | - GPU used: 1x NVIDIA GeForce GTX 1080Ti |
| | - Training time: 21 hours, 4 minutes, and 57 seconds |
| |
|
| | # Training Loss |
| | ``` |
| | Step Training Loss |
| | 100 0.330900 |
| | 200 0.293000 |
| | 300 0.276500 |
| | 400 0.290900 |
| | 500 0.306100 |
| | 600 0.302600 |
| | 700 0.337200 |
| | 800 0.295000 |
| | 900 0.297800 |
| | 1000 0.299500 |
| | 1100 0.268900 |
| | 1200 0.257800 |
| | 1300 0.264100 |
| | 1400 0.294400 |
| | 1500 0.293900 |
| | 1600 0.287600 |
| | 1700 0.281200 |
| | 1800 0.273400 |
| | 1900 0.266600 |
| | 2000 0.227500 |
| | 2100 0.261600 |
| | 2200 0.275700 |
| | 2300 0.290100 |
| | 2400 0.290900 |
| | 2500 0.316200 |
| | 2600 0.296500 |
| | 2700 0.291400 |
| | 2800 0.253300 |
| | 2900 0.321500 |
| | 3000 0.269500 |
| | 3100 0.295600 |
| | 3200 0.265800 |
| | 3300 0.262800 |
| | 3400 0.274900 |
| | 3500 0.259800 |
| | 3600 0.226300 |
| | 3700 0.325700 |
| | 3800 0.249000 |
| | 3900 0.237200 |
| | 4000 0.251400 |
| | 4100 0.247000 |
| | 4200 0.278700 |
| | 4300 0.264000 |
| | 4400 0.245000 |
| | 4500 0.235900 |
| | 4600 0.240400 |
| | 4700 0.235200 |
| | 4800 0.220300 |
| | 4900 0.202700 |
| | 5000 0.240500 |
| | 5100 0.258500 |
| | 5200 0.236300 |
| | 5300 0.267500 |
| | 5400 0.236700 |
| | 5500 0.265900 |
| | 5600 0.244900 |
| | 5700 0.297900 |
| | 5800 0.281200 |
| | 5900 0.313800 |
| | 6000 0.249800 |
| | 6003 0.271939 |
| | ``` |
| |
|
| | # Example Usage |
| | ```py |
| | from transformers import BitsAndBytesConfig, AutoTokenizer, AutoModelForCausalLM |
| | import torch |
| | import accelerate |
| | |
| | use_4bit = True |
| | bnb_4bit_compute_dtype = "float16" |
| | bnb_4bit_quant_type = "nf4" |
| | use_double_nested_quant = True |
| | compute_dtype = getattr(torch, bnb_4bit_compute_dtype) |
| | |
| | # BitsAndBytesConfig 4-bit config |
| | bnb_config = BitsAndBytesConfig( |
| | load_in_4bit=use_4bit, |
| | bnb_4bit_use_double_quant=use_double_nested_quant, |
| | bnb_4bit_quant_type=bnb_4bit_quant_type, |
| | bnb_4bit_compute_dtype=compute_dtype, |
| | load_in_8bit_fp32_cpu_offload=True |
| | ) |
| | |
| | # Load model in 4-bit |
| | tokenizer = AutoTokenizer.from_pretrained("AlfredPros/CodeLlama-7b-Instruct-Solidity") |
| | model = AutoModelForCausalLM.from_pretrained("AlfredPros/CodeLlama-7b-Instruct-Solidity", quantization_config=bnb_config, device_map="balanced_low_0") |
| | |
| | # Make input |
| | input='Make a smart contract to create a whitelist of approved wallets. The purpose of this contract is to allow the DAO (Decentralized Autonomous Organization) to approve or revoke certain wallets, and also set a checker address for additional validation if needed. The current owner address can be changed by the current owner.' |
| | |
| | # Make prompt template |
| | prompt = f"""### Instruction: |
| | Use the Task below and the Input given to write the Response, which is a programming code that can solve the following Task: |
| | |
| | ### Task: |
| | {input} |
| | |
| | ### Solution: |
| | """ |
| | |
| | # Tokenize the input |
| | input_ids = tokenizer(prompt, return_tensors="pt", truncation=True).input_ids.cuda() |
| | # Run the model to infere an output |
| | outputs = model.generate(input_ids=input_ids, max_new_tokens=1024, do_sample=True, top_p=0.9, temperature=0.001, pad_token_id=1) |
| | |
| | # Detokenize and display the generated output |
| | print(tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True)[0][len(prompt):]) |
| | ``` |
