jaandoui / DNABERT2-AttentionExtracted

huggingface.co
Total runs: 1.6K
24-hour runs: -70
7-day runs: 267
30-day runs: 1.0K
Model's Last Updated: May 19 2024

Introduction of DNABERT2-AttentionExtracted

Model Details of DNABERT2-AttentionExtracted

This version of DNABERT2 has been changed to be able to output the attention too, for attention analysis.

To the author of DNABERT2, feel free to use those modifications.

Use --model_name_or_path jaandoui/DNABERT2-AttentionExtracted instead of the original repository to have access to the attention.

Most of the modifications were done in Bert_Layer.py. It has been modified especially for fine tuning and hasn't been tried for pretraining. Before or next to each modification, you can find "JAANDOUI" so to see al modifications, search for "JAANDOUI" . "JAANDOUI TODO" means that if that part is going to be used, maybe something might be missing.

Now in Trainer (or CustomTrainer if overwritten) in compute_loss(..) when defining the model: outputs = model(**inputs, return_dict=True, output_attentions=True) activate the extraction of attention: output_attentions=True (and return_dict=True (optional)). You can now extract the attention in outputs.attentions Note than the output has a third dimension, mostly of value 12, referring to the layer outputs.attentions[-1] refers to the attention of the last layer. Read more about model outputs here: https://huggingface.co/docs/transformers/v4.40.2/en/main_classes/output#transformers.utils.ModelOutput

I'm also not using Triton, therefore cannot guarantee that it will work with it.

I also read that there were some problems with extracting attention when using Flash Attention here: https://github.com/huggingface/transformers/issues/28903 Not sure if that is relevant for us, since it's about Mistral models.

I'm still exploring this attention, please don't take it as if it works 100%. I'll update the repository when I'm sure.

The official link to DNABERT2 DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome .

READ ME OF THE OFFICIAL DNABERT2: We sincerely appreciate the MosaicML team for the MosaicBERT implementation, which serves as the base of DNABERT-2 development.

DNABERT-2 is a transformer-based genome foundation model trained on multi-species genome.

To load the model from huggingface:

import torch
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("zhihan1996/DNABERT-2-117M", trust_remote_code=True)
model = AutoModel.from_pretrained("zhihan1996/DNABERT-2-117M", trust_remote_code=True)

To calculate the embedding of a dna sequence

dna = "ACGTAGCATCGGATCTATCTATCGACACTTGGTTATCGATCTACGAGCATCTCGTTAGC"
inputs = tokenizer(dna, return_tensors = 'pt')["input_ids"]
hidden_states = model(inputs)[0] # [1, sequence_length, 768]

# embedding with mean pooling
embedding_mean = torch.mean(hidden_states[0], dim=0)
print(embedding_mean.shape) # expect to be 768

# embedding with max pooling
embedding_max = torch.max(hidden_states[0], dim=0)[0]
print(embedding_max.shape) # expect to be 768

Runs of jaandoui DNABERT2-AttentionExtracted on huggingface.co

1.6K
Total runs
-70
24-hour runs
61
3-day runs
267
7-day runs
1.0K
30-day runs

More Information About DNABERT2-AttentionExtracted huggingface.co Model

DNABERT2-AttentionExtracted huggingface.co

DNABERT2-AttentionExtracted huggingface.co is an AI model on huggingface.co that provides DNABERT2-AttentionExtracted's model effect (), which can be used instantly with this jaandoui DNABERT2-AttentionExtracted model. huggingface.co supports a free trial of the DNABERT2-AttentionExtracted model, and also provides paid use of the DNABERT2-AttentionExtracted. Support call DNABERT2-AttentionExtracted model through api, including Node.js, Python, http.

DNABERT2-AttentionExtracted huggingface.co Url

https://huggingface.co/jaandoui/DNABERT2-AttentionExtracted

jaandoui DNABERT2-AttentionExtracted online free

DNABERT2-AttentionExtracted huggingface.co is an online trial and call api platform, which integrates DNABERT2-AttentionExtracted's modeling effects, including api services, and provides a free online trial of DNABERT2-AttentionExtracted, you can try DNABERT2-AttentionExtracted online for free by clicking the link below.

jaandoui DNABERT2-AttentionExtracted online free url in huggingface.co:

https://huggingface.co/jaandoui/DNABERT2-AttentionExtracted

DNABERT2-AttentionExtracted install

DNABERT2-AttentionExtracted is an open source model from GitHub that offers a free installation service, and any user can find DNABERT2-AttentionExtracted on GitHub to install. At the same time, huggingface.co provides the effect of DNABERT2-AttentionExtracted install, users can directly use DNABERT2-AttentionExtracted installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

DNABERT2-AttentionExtracted install url in huggingface.co:

https://huggingface.co/jaandoui/DNABERT2-AttentionExtracted

Url of DNABERT2-AttentionExtracted

DNABERT2-AttentionExtracted huggingface.co Url

Provider of DNABERT2-AttentionExtracted huggingface.co

jaandoui
ORGANIZATIONS